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For  the  last  three  years  the  Military  Training  Centers  (MTC) 

PLATO  support  group  has  been  experimenting  with  the  development  of  a 
number  of  formalized  techniques  for  assisting  authors  in  writing  instruc- 
tionally  effective  courseware.  These  methods,  collectively  referred  to 
as  "lesson  reviewing",  vary  widely  depending  on  individual  authors' 
needs.  Basically,  however,  a lesson  review  is  a set  of  comments  about  the 
lesson  ranging  from  alternative  instructional  strategy  suggestions  to 
grammatical  corrections,  from  content  accuracy  to  punctuation  (see 
Lesson  Review,  MTC  report  #3,  1976,  Larry  Francis,  Merle  Goldstein,  and 
Eileen  Call-Himwich  for  examples  and  complete  descriptions  of  review 
techniques) . 

This  report  describes  the  evolution  of  the  lesson  review  process 

and  discusses  review  effectiveness  in  terms  of  resulting  lesson  revision 

and  author  training.  The  report  also  details  a number  of  recommendations 

for  improving  review  effectiveness  in  the  future.  The  following  areas: 

— reviewer  characteristics 
— reviewer/author  relationship 
— review  content /format 

are  emphasized.  Some  specific  recommendations  are  that  the  reviewer 
should  be  on-site,  that  the  reviewer's  actual  or  perceived  authority 
should  be  equal  to  the  author's,  and  that  the  reviewer  and  author  should 
reach  either  a contractual  or  lesson  formalized  agreement  about  the  purpose 
of  the  review  and  about  what  specific  feedback  each  expects. 
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Preface 

Due  to  the  personal  nature  of  critiquing  lessons,  data  for  the 
following  report  is  largely  anecdotal.  Some  information  has  also  been 
gathered  through  the  questionnaire  in  Appendix  A or  in  face  to  face  or 
phone  interviews.  During  the  course  of  our  "reviewing"  experiments  and 
evaluation  work,  we  have  observed  a marked  inconsistency  between  what 
authors  say  and  what  they  do.  Some  authors  replied  with  nothing  but 
praise  both  to  reviews  themselves  and  to  the  questionnaire,  but  had 
rarely  responded  to  anything  more  than  the  most  rudimentary  recommendations 
in  reviews.  This  dichotomy  proved  to  be  our  nemesis  for  quite  a time 
before  we  finally  began  hearing  the  "metatalk"  underneath  the  words — 
what  was  really  being  communicated  rather  than  the  words  that  were  being 
said. 


r . -- 

Introduction 
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[ For  the  last  three  years  the  Military  Training  Centers  (MTC) 

I support  group  has  been  experimenting  with  the  development  of  a number  of 

formalized  techniques  for  assisting  authors  in  writing  courseware  that  is 
[ more  instructionally  effective.  These  methods,  collectively  referred  to 

as  "lesson  reviewing",  vary  widely  depending  on  individual  authors' 

; ^ needs.  Basically,  however,  a lesson  review  is  a set  of  comments  about  the 

lesson  ranging  from  alternative  instructional  strategy  suggestions  to 
grammatical  corrections,  from  content  accuracy  to  punctuation. 

Lesson  reviewing  was  originally  conceived  for  the  dual  purpose  of: 

1.  aiding  authors  in  lesson  revision 
2.  training  authors  in  instructional  design. 


The  purpose  of  the  following  report  is  to  both  describe  the  evolution 
of  the  lesson  review  process  and  discuss  review  effectiveness  in  terms  of 
lesson  revision  and  author  training. 


Rationale 
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When  the  MTC  group  first  began  reviewing  lessons,  the  rationale 
behind  the  need  for  reviews  was  grounded  in  a threefold  problem.  First, 
while  a number  of  ARPA  authors  had  teaching  experience,  few  were  familiar 
with  either  computer-based  education  or  instructional  design.  Many 
authors  found  the  transition  from  classroom  to  computer  both  perplexing 
and  frustrating.  What  once  could  have  been  taught  in  a lecture  format 
now  required  a more  innovative  approach.  Instructors  who  could  once 
comfortably  rely  on  standardized  course  objectives  and  lists  were  suddenly 
wrestling  with  the  subtleties  of  student  behavioral  objectives  and  credible 
criterion  testing.  In  addition,  since  the  MTC  group  offered  a two  to 
three  week  course  in  both  the  TUTOR  language  and  the  rudiments  of 
instructional  strategy,  many  authors  felt  somewhat  harried  over  having 
to  learn  TUTOR,  Instructional  design,  and  the  mechanics  of  the  PLATO 
keyset  and  editor  all  at  one  time.  Part  of  that  frustration  was  the 
sometimes  stated,  more  often  implied  conviction  that  instructional 
design  seminars  "interrupted"  the  "more  important"  task  of  learning 
TUTOR.  Having  trained  a number  of  diverse  author  groups,  we've  found 
the  premise  that  instructional  strategy  is  basically  intuitive  and  that 
authors  need  little  if  any  assistance  in  lesson  design  to  be  a particu- 
larly persistent,  widely  held  attitude.  This  is  especially  true  of  former 
instructors  who  sometimes  feel  that  knowledge  of  one  medium  implies 
knowledge  of  another  medium,  i.e.  success  in  a lecture  situation  Implies 
"knowing  how  to  teach"  using  an  approach  even  as  dissimilar  and  indivi- 
dualized as  CBE. 
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Another  element  of  author  frustration,  however,  was  also  the  problem 
of  being  bombarded  with  a variety  of  new  concepts  simultaneously.  Thus, 
the  MTC  group  saw  lesson  reviewing  as  one  method  of  continuing  training 
even  after  authors  had  returned  to  their  sites. 

Finally,  even  experienced  authors  can  benefit  from  an  outside, 
objective  appraisal  of  their  work,  particularly  in  those  areas  not 
readily  analyzable  from  student  performance  data,  i.e.  amount  of  inter- 
action, organization,  alternative  approaches,  visual  presentation,  etc. 

Lesson  reviewing,  though  by  no  means  ideal,  seemed  a potentially 
effective  way  of  handling  all  three  situations. 

Evolution  of  the  Reviewing  Process 

End-of-lesson  reviews.  Although  MTC  did  a few  early  reviews  at 
Chanute,  the  first  formalized  reviews  were  written  for  the  Aberdeen 
Machinist  Course  Project  at  Aberdeen  Proving  Grounds.  By  the  end  of  the 
Aberdeen  project,  approximately  fifteen  MTC  reviews  had  been  written. 

These  early  reviews,  later  referred  to  as  "end-of-lesson"  reviews,  were 
often  a combination  of  coding  and  instructional  design  suggestions  with 
the  main  emphasis  on  the  latter.  While  the  tone  and  slant  of  each  review 
certainly  varied  with  individual  reviewers,  end-of-lesson  reviews  all 
had  a number  of  common  characteristics. 

1.  They  were  always  written  after  the  lesson  had  been  completed. 

2.  They  were  always  rendered  in  written  form,  frequently 
including  an  annotated  printout  of  the  lesson  suggesting 
coding  changes,  textual  revisions  etc. 

3.  They  generally  required  from  one  to  two  months  to  be  written 
and  delivered  to  the  author. 
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— planning 
— design 

— Implementation 
— polish 

Each  level  In  turn  spanned  a variety  of  possible  problem  areas  and 
recommendations  or  suggestions  for  alternatives.  The  following  examples 
detail  many  of  the  sorts  of  Issues  that  were  dealt  with. 

Level  One — Planning 

— Intentions,  goals,  objectives  of  the  lesson 
— assumptions  about  students'  entering  ability  and  knowledge 
of  the  subject  (e.g.  terminology) 

— the  relationship  between  the  Individual  lesson  and 
the  overall  curriculum 

— the  relevance  of  Individual  topics  to  the  lesson  as  a whole 

Level  Two — Design 

— choice  of  teaching  strategies 
— selection/appropriateness  of  media 
— individualized  routing 
— organization  of  content 

Level  Three — Implement at ion/Development 

— effective  utilization  of  teaching  strategies  and  medium 
— corrective  feedback  and  remediation 
— appropriateness  of  tone  and  style 
— reliability  of  the  criterion  test  (if  present) 

— transition  from  frame  to  frame,  topic  to  topic 
— quality  and  quantity  of  student  interaction 
— clarity  of  text 

— appropriateness  of  reading  level,  illustrations,  etc. 

Level  Four — Polish 

— grammar,  spelling,  typographical  errors,  etc. 

— consistency  of  terminology,  instructions,  keys,  etc. 

— visual  presentation  (Including  textual  layout) 

Reviews  were  generally  formated  to  include  the  following  elements: 
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1.  a cover  letter  enumerating  the  major  points  brought  out 
in  the  body  of  the  review 

2.  approximately  six  pages  (reviews  ranged  from  4-11  pages) 
of  general  comments  and  suggested  changes 

3.  flowcharts  of  both  the  present  lesson  structure  and  suggested 
organizational  revisions 

4.  an  annotated  printout  of  the  program  itself  with  line-by-line 
textual  or  coding  suggestions. 

For  a complete  description  of  Aberdeen  lesson  development,  reviews  and 
courseware  see  Summary  and  Analysis  of  Aberdeen  CBE  Project  (Call-Himwich , 
1977;  Himwich,  1977). 

While  Aberdeen  authors  usually  claimed  that  the  comments  and  suggestions 
they  received  in  reviews  were  "useful"  and  "helpful",  they  also  seemed 
inclined  to  simply  file  reviews  away  and  forget  about  them.  MTC 
reviewers  conceded  that  they  generally  had  scant  success  with  getting 
authors  to  make  anything  other  than  superficial  lesson  revisions  (i.e. 
correcting  misspellings,  etc.).  In  addition  it  was  often  difficult  to 
get  authors  to  comment  about  the  reviews  themselves  with  anything  more  than 
a perfunctory  "thank  you".  fiTC  reviewers  felt  that  definite  changes  in 
revie  ? needed  to  be  made.  The  establishment  of  the  Sheppard  project 
presented  an  excellent  opportunity  to  experiment  with  alternative  review 
methods. 

Reviewing  pitfalls.  Hampered  by  a lack  of  author  feedback,  MTC 
reviewers  were  forced  to  draw  their  own  conclusions  about  the  new 
form  or  direction  reviews  should  take.  They  began  by  assuming  that  the 
quality  of  the  reviews  was  not  in  question,  but  rather  that  other  more 
subjective  or  ambiguous  elements  were  involved.  The  fact  that  authors 
were  so  reluctant  to  talk  about  reviews  seemed  to  support  the  assumption 
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that  feelings  rather  than  issues  were  at  stake.  Thus,  MTC  turned  its 
attention  from  revising  review  comments  to  revamping  review  techniques , 
and  from  the  Impact  reviews  had  on  lessons  to  the  effect  they  had  on 
authors.  In  emphasizing  the  subjective  elements  of  reviews,  reviewers 
set  about  trying  to  pinpoint  the  possible  technical  or  psychological 
pitfalls  in  effect  at  the  time  a lesson  was  reviewed. 

Psychological  Elements.  From  MTC's  perspective,  three  psychological 
factors  played  heavily  in  the  acceptance  or  rejection  of  a lesson  review. 
First  and  most  obvious  was  the  problem  of  the  author's  subjective  involve- 
ment in  the  lesson.  As  one  (male)  author  effused,  "Writing  lessons  is  a 
very  creative,  difficult  thing.  It’s  like  giving  birth."  Thus,  just  as 
artists  sometimes  find  it  difficult  to  separate  themselves  from  their  work, 
authors  often  find  it  difficult  to  separate  themselves  from  their  lessons. 
To  criticize  the  lesson  was  to  criticize  the  author.  This  was  especially 
true  if  the  author  and  reviewer  had  never  met.  In  this  Instance  the 
reviewer  was  often  regarded  more  as  an  adversary  than  an  ally,  the  review 
itself  more  as  a threat  than  a tool. 

Apart  from  the  question  of  subjectivity,  there  was  the  problem  of 
time.  A thorough  end-of-lesson  review  typically  required  a number  of 
weeks  to  prepare.  During  that  time  the  author  had  usually  begun  work  on  a 
new  lesson.  Since  enthusiasm  for  the  old  lesson  was  usually  replaced  by 
the  preoccupation  and  momentum  of  work  on  the  new  lesson,  revision  was 
often  relegated  to  a limbo  status  to  be  carried  out  "as  soon  as  this  new 
lesson  in  finished".  As  the  old  lesson  got  "colder",  rev  ’iun  seemed 
less  and  less  important  until  it  was  easiest  to  regard  a lesson  as 
"finished"  simply  because  coding  was  complete. 


A 
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Perhaps  the  most  detrimental  drawback  of  end-of-lesson  reviews. 

•> 

' however,  was  the  problem  of  the  part  versus  the  whole.  Any  task  is  more 

i 

burdensome  viewed  as  a whole  rather  than  in  parts.  Singly,  each 
criticism  or  suggestion  a reviewer  made  iT.ay  have  seemed  reasonable. 

! Collectively,  however,  the  suggested  chanees  may  have  been  overwhelming, 

j After  receiving  a printout  annotated  in  red  ink,  one  author  ruefully 

observed,  "It  looks  like  it  bled  to  death."  He  also  chose  to  ignore  it. 

' Technical  Elements.  The  two  major  technical  problems  we  perceived 

‘ centered  around  one  outstanding  dilemma — time.  At  the  ARPA  sites  we 

worked  with,  authors  were  under  considerable  pressure  to  meet  semester 
dates,  project  deadlines,  etc.  Thus,  they  sometimes  felt  an  understand- 
able reluctance  to  "waste  time"  revising  lessons  which  were  essentially 
considered  finished. 

In  addition,  reviewers  sometimes  found  themselves  suggesting  changes 
which  would  have  required  a major  overhaul.  The  need  for  such  substantial 
revision  might  well  have  been  averted  had  the  author  and  reviewer  been 
able  to  consult  in  the  planning  stages,  using  the  review  as  a proposal 
rather  than  a post  mortem. 

The  problem  of  author/reviewer  consultation  was  also  a stumbling 
block.  Almost  by  definition,  an  end-of-lesson  review  was  a lengthy, 
one-sided  excursion  through  the  lesson's  strengths  and  shortcomings.  This 
monologue  quality,  coupled  with  the  other  inhibitiva  elements,  probably  did 
little  to  enhance  any  real  exchange  of  ideas  between  author  and  reviewer. 

In-progress  reviewing.  With  the  onset  of  the  Sheppard  AFB  paramedical 
project,  MTC  felt  that  the  time  was  right  for  experimenting  with  different 
review  techniques.  The  main  problem,  was  to  develop  a flexible  process  that 


could : 
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1.  reduce  author  defensiveness  by  examining  the  lesson  before 
the  author's  subjective  involvement  became  too  strong 


capitalize  on  lesson  momentum  by  giving  the  author  immediate 
feedback  while  the  lesson  was  still  "current" 


3.  optimize  author  time  by  enabling  the  author  to  make  any 
necessary  modifications  or  corrections  before  they  became 
habitual . 


The  most  reasonable  first  step  seemed  to  be  to  begin  reviewing  lessons 
as  they  were  being  written,  in  bits  and  pieces  and  at  various  stages  of 
development.  To  maintain  continuity  and  an  overall  perspective,  In-progress 
reviewing  was  intended  as  a cumulative  (rather  than  fragmentary)  approach 
in  which  the  reviewer  would  reexamine  "old"  sections  of  a lesson  in 
addition  to  each  successive  "new"  section.  Since  the  reviewer  had  only  to 
deal  with  part  of  a lesson  at  a time,  review  comments  could  be  written  much 
more  quickly.  In  order  to  provide  immediate  feedback  we  also  established 
an  on-line  "review"  file  in  which  reviewers  wrote  general  comments,  and 
authors  were  encouraged  to  make  replies  or  rebuttals,  ask  questions,  etc. 
Additional  more  specific  comments  were  then  sent  somewhat  later  (usually 
within  the  week)  in  a hardcopy  form  along  with  a lesson  printout  mainly 
noting  suggestions  for  textual  changes,  alternative  wordings,  etc.  The  only 
problem  was  that  since  the  author  was  continually  revising  and  expanding  the 
lesson  the  ha’'dcopy  comments  were  sometimes  outdated  by  the  time  they 
reached  him  (her) . 


The  development  of  PLATO  inter-terminal  communication  and  monitoring 


capabilities  also  added  another  dimension  to  reviewing.  It  was  now 


This  PLATO  communications  feature  enables  two  people  at  two  different 
terminals  to  go  through  a lesson  "together".  One  person  enters  a lesson, 
presses  certain  keys,  and  the  lesson  appears  simultaneously  on  the  second 
terminal.  Any  responses  entered  into  the  first  terminal  also  cause  the 
display  on  the  second  terminal  to  react  Identically.  Authors  are  also  able 
to  "talk"  to  each  other  on-line  while  in  this  monotoring  framework. 


1 
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possible  for  tlie  author  and  the  reviewer  to  be  at  different  sites  and  yet 
go  through  a lesson  together  while  simultaneously  being  able  to  "talk" 
witli  each  other.  Since  the  reviewer  usually  had  specific  comments  in  mind 
about  a number  of  different  sections  of  the  lesson,  the  author  usually 
monitored  the  reviewer  so  that  the  reviewer  could  direct  their  progress 
through  the  lesson.  This  method  not  only  made  it  easier  for  each  to 

; 

explain  various  comments  or  thoughts  about  the  lesson,  but  also  enriched 

I 

the  review  as  more  an  exchange  of  ideas  and  less  a set  of  directives  given  ! 

j 

over  to  the  author.  While  many  authors  favored  this  method,  it  was  extremely  j 

time  consuming,  ranging  from  l'-^  to  nearly  3 hours  in  length,  and  reqi; iring  I 

! 

a good  deal  of  mental  stamina  on  the  part  of  both  the  author  and  the  I 

reviewer. 

In-progress  reviews  covered  essentially  the  same  areas  and  levels  as 
end-of-lesson  reviews.  Whereas  end-of-lesson  reviews  tended  to  be  more 
lesson  oriented,  reviewers  tried  to  make  in-progress  reviews  more  au,..ior 
oriented.  Written  comments  were  always  prefaced  with  a recitation  oi  what 
the  reviewer  had  especially  liked  about  the  lesson,  i.e.  "The  graphics 
in  unit  x are  especially  effective",  or  "You  generally  provide  helpful 
feedback  for  incorrect  answers".  Reviewers  also  tried  to  intersperse 

! 

remaining  comments  with  specific  examples  not  only  of  what  could  be 
changed  but  what  was  good.  Wherever  possible  the  reviewer  tried  to 
reference  one  part  of  a lesson  and  apply  it  as  a solution  to  another  part, 
e.g.  "This  section  might  be  more  easily  illustrated  by  using  the  same  sort 

of  graph  you  used  in  section  ". 

By  the  end  of  the  Sheppard  paramedical  project,  MTC  reviewers  had 
performed  over  100  lesson  reviews  for  Sheppard  authors.  For  a thorough 
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1 description  of  MTC  review  techniques  with  sample  reviews  see  Lesson  Review 

(Francis,  Goldstein,  & Call-Hiniwich,  1975). 

■» 

i 


; Conclusion 

« 

I *• 

' / Having  experimented  with  a number  of  reviewing  approaches,  we  find 

ourselves  in  some  senses  boggled  by  even  more  hypotheses  than  when  we 
began.  As  is  so  often  the  case,  the  more  one  learns,  the  more  there  is 
to  be  learned.  Our  increasing  understanding  of  the  subtleties  of  working 
simultaneously  on  an  intellectual  and  emotional  level  has  left  us  with 
' more  questions  than  answers.  Therefore,  the  following  section  enumerates 

' • not  only  what  we  know  but  what  we  don't  know. 

The  conclusions  fall  into  two  categories — what  was  accomplished  and 
what  might  be  accomplished.  Topics  in  the  first  category  include  assess- 
ments of  review  effectiveness  in  regard  to  MTC's  dual  goal  with  related 
incidents  and  author  comments.  Topics  in  the  second  classification  center 
on  recommendations  for  improving  review  effectiveness  in  the  future. 

Before  examining  the  relative  "success"  of  each  type  of  reviewing, 
some  concomitant  philosophical  Issues  should  be  explored.  Two  major 

r 

; quandaries  plagued  reviewers  (and  probably  authors)  throughout  the  course 

of  critiquing  a project's  courseware.  What  is  the  relationship  between 

j author  and  reviewer?  What  is  the  purpose  of  the  review?  Obviously  these 

t 

two  questions  are  interrelated.  Very  often  the  rapport  between  author 

! and  reviewer  determines  the  purpose  or  Importance  of  the  review.  This  is 

exactly  the  problem.  In  our  experience,  neither  the  reviewer's  authority 
nor  responsibility  was  ever  clearly  defined.  As  a result,  reviewers 
generally  felt  that  suggested  lesson  changes  that  were  Instituted  were 
usually  a result  of  whatever  powers  of  persuasion  or  cajolery  they  could 
summon.  While  the  reviewer  had  no  actual  authority,  s(he)  usually  felt 


u 


” some  ethical  responsibility  for  the  quality  of  the  lesson.  Consequently, 

i 

reviewers  found  themselves  continually  vexed  with  the  question  of  whether 

i. 

their  responsibility  ended  with  aiding  authors  to  meet  the  author's  own 
standards  or  whether  they  (reviewers)  should  try  to  entice  the  author  into 
conforming  to  their  standards.  Usually  they  tried  to  aim  somewhere 

! 

' between  these  two  points,  relenting  in  some  areas  while  holding  to  others. 

I Authors  sometimes  complained  about  the  "university  approach"  to 

i 

■ lesson  writing  claiming  that  military  students  were  "different",  so 

certain  Instructional  approaches  (increased  interaction,  open-ended 
questions,  etc.)  were  neither  applicable  nor  feasible.  It  is  true 
that  there  are  differences  between  a military  and  university  setting. 
Aberdeen  authors  were  writing  lessons  for  a widely  varied  group  ranging 
from  high  school  dropouts  to  college  students.  Also,  the  course  material 
required  very  few  cognitive  skills,  centering  largely  on  mechanical 
skills  required  to  machine  various  sorts  of  keys,  keyways,  gear  teeth, 
etc.  As  one  author  wrote,  "It  is  a drastic  change  to  step  from  the  Univer- 
sity of  Illinois  into  an  army  classroom.  Idealism  and  the  university 
approach  can  be  disastrous."  It  is  true  that  some  aspects  of  "technical 
training"  differ  from  academics,  and  reviewers  gradually  began  to  phase 
out  comments  in  some  areas.  However,  many  techniques  MTC  suggested  wliich 
were  labeled  "university  approaches"  were  actually  strategies  recommended 
or  mandated  by  instructional  design  guidelines  and  rules  for  the  individual 
armed  services.  Most  so-called  "university  approaches"  were  never  shown  to 
either  succeed  or  fail  since  authors  often  refused  to  try  them  saying, 

"That  would  never  work  with  our  students."  After  a while  we  began  to 


understand  that  very  often  citing  "the  difference"  was  really  a way  of 
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I 

I . saying,  "I  don't  like  that",  or  "I  don't  want  to  do  that."  In  one 

! *'■  instance  reviewers  suggested  that  authors  at  one  ARPA  site  try  a 

I ' 

visual  technique  that  had  originated  at  another  ARPA  site.  In  this 
instance  also  authors  at  the  first  site  claimed  that  their  students 
were  "different"  so  such  a technique  was  neither  "necessary"  nor  desirable. 
Reviewers  began  to  conclude  that  something  other  than  "the  university 
approach"  might  actually  be  the  problem.  Some  authors  even  implied  that 
the  problem  was  a matter  of  defensiveness,  of  feeling  that  the  second 
site  was  getting  praise  from  reviewers  while  the  first  was  mainly  getting 
criticism. 

Consequently,  reviewers  felt  trapped  between  two  conflicting  convic- 
tions. While  they  believed  that  lesson  quality  was  mainly  the  author's 
responsibility , they  also  felt  that  falling  to  comment  on  a serious  or 
recurring  problem  might  be  misinterpreted  as  condoning  or  even  encouraging 
something  the  reviewer  in  fact  regarded  as  ineffective  or  even  detrimental. 

What  Was  Accomplished 

End-of-lesson  reviews.  Viewed  with  the  inevitable  clarity  of 

hindsight,  the  end-of-lesson  review  technique  was  almost  a complete 

failure  both  as  an  author  aid  and  as  a training  tool.  Authors  mainly 

made  only  minor  punctuation,  spelling,  or  working  changes,  or  coding  changes 

(though  reviews  didn't  usually  stress  coding  revision).  In  a number  of 

2 

instances  even  execution  errors  which  were  pointed  out  in  a review  went 


Execution  errors  are  programming  errors  so  serious  that  the  lesson 
will  cease  to  function  when  the  student  gets  to  that  point  within  the 
' son.  Typically,  the  student  is  then  taken  to  a display  that  says, 
ihere's  an  error  in  this  lesson.  Try  another  one"  and  then  rerouted  back 
to  a course  index. 

I 

i 

t 


i 
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unchanged.  As  one  author  wrote,  "I  changed  coding  but  fought  most 
suggested  changes  to  our  instructional  design  plan."  When  asked  what 
the  main  advantages  of  reviews  were  another  author  wrote,  "...  help 
with  coding — corrections,  a shorter,  better  way  to  code,  etc."  One 
explanation  is  that  coding  changes  are  easier  to  make  than  strategy 
changes.  As  one  author  said,  "We  tend  to  get  involved  with  the  computer 
and  forget  the  desired  result,  training  the  students." 

Author  comments  about  the  deficiencies  of  end-of-lesson  reviews 
center  on  three  synergetic  problem  areas.  First,  authors  commented  that 
reviews  were  too  detailed  (a  problem  shared  by  in-progress  reviews) . 

One  author  commented  "Your  attention  to  detail  will  destroy  a new, 
marginal  or  struggling  author."  This  partially  supported  our  conclusion 
that  the  sheer  numbers  of  comments  resulting  from  reviewing  a lesson  in 
its  entirety  could  be  devastating.  A second  and  perhaps  more  serious 
problem  was  lack  of  communication  that  resulted  not  only  because  of  the 
divergence  between  "professional"  and  "lay"  people,  but  between  civilian 
and  military  approaches.  One  author  commented,  "Your  wording  of  changes 
and  new  coding  was  very,  very  poor  for  those  programming  1,000  miles  away," 
while  another  author  wrote,  "(an  ex-military  MTC  member)  was  best  able  to 
review  [our]  lessons  because  he  could  speak  our  language.  The  majority  of 
the  other  reviewers  were  good  but  the  communication  gap  was  real  difficult 
to  cross."  Yet  contrary  to  this  claim,  even  this  ex-military  reviewer's 
comments  went  largely  unheeded.  A third,  related  theme  that  reverberated 

I 

1 

I 


throughout  nearly  every  ARPA  project  was  the  element  of  culture  shock 
resulting  from  civilians  and  military  personnel  working  in  a cooperative 
effort.  When  asked  to  rate  the  importance  of  a number  of  areas  included 
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in  reviews,  one  author  wrote  with  reference  to  content  accuracy,  'In  our 
case  you  could  not  help,  even  in  areas  where  you  all  were  trained  i.e. 
test  construction.  The  Idealistic  university  approach  prevailed."  Since 
one  of  the  two  (at  that  time)  MTC  reviewers  held  a master's  degree  in 
instructional  design,  and  since  many  suggestions  were  taken  directly 
from  Air  Force  Instructional  design  regulations,  such  a criticism  did 
not  seem  valid.  Furthermore,  the  review  recommendations  were,  once 
again,  never  tried  and  proven  effective  or  ineffective.  As  one  author 
said,  "The  authors  never  gave  the  reviewers  a fair  chance." 

In-progress  reviews.  In-progress  review  techniques  were  considerably 
more  successful  in  effecting  short  range  lesson  changes,  but  only  marginally 
successful  as  a long  range  training  device.  Authors  instituted  an  average 
of  50-75%  of  the  changes  suggested  in  MTC  reviews.  These  changes  consisted 
mainly  of  adding  graphics  to  replace  or  clarify  text,  adding  various  types 
of  questions,  improving  existing  questions,  distilling  wordy  text,  rear- 
ranging or  breaking  up  heavily  texted  displays,  punctuation,  spelling, 
etc.  The  relative  "success"  of  in-progress  reviewing  was  due  in  part  to  the 
rapport  built  up  through  almost  daily  communication  between  site  authors 
and  MTC  reviewers.  The  "success"  of  in-progress  reviews,  however,  also 
supports  our  assumption  that  criticism  should  be  doled  out  a little  at  a 
time. 

This  is,  of  course,  not  to  claim  that  in-progress  reviews  are  free 
of  snags.  A number  of  issues  still  remain  unsolved.  For  example,  since 
there  were  only  two  MTC  reviewers  who  performed  in-progress  reviews,  it 
seemed  to  evolve  that  authors  usually  had  lessons  critiqued  by  the  same 
reviewer  each  time.  Working  on  successive  lessons  together  of  course 
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resulted  in  authors  and  reviewers  building  up  a certain  rapport.  Thus, 
the  author  came  to  know  what  sorts  of  comments  that  particular  reviewer 

( 

was  liable  to  make,  and  the  reviewer  came  to  understand  what  sorts  of 
comments  would  be  accepted  and  what  would  not.  As  a result  some  comments 

1 

! were  eventually  dropped  entirely  or  at  best  categorized  as  perpetual 

' debates.  For  example,  one  author  and  reviewer  continually  debated  whether 

the  author's  lesson  material  actually  belonged  on  PLATO  or  on  a printed 

I 

' handout.  The  reviewer  would  say,  "Just  for  the  record,  my  old  objection 

still  stands",  and  the  author  would  acknowledge  the  comment  and  then 
ignore  it.  Both  knew  that  the  suggestion  would  be  made  and  that  it  would 
be  ignored. 

Another  more  serious  problem  was  the  author  dependency  that  sometimes 
resulted  for  reasons  which  are  not  entirely  clear  even  now.  Some  authors 
relied  heavily  on  lesson  reviews  as  the  sole  "validation"  of  a lesson.  For 
example,  in  spite  of  repeated  promptings,  one  author  insisted  on  relying 
on  reviewers  to  find  problems  in  the  lesson  rather  than  running  students 
through  the  lesson  before  it  was  given  to  actual  project  students.  When 
actual  students  did  take  his  lessons  for  the  first  time,  they  naturally 
found  a number  of  errors  that  reviewers  had  not  been  able  to  anticipate. 
Rather  than  proving  the  need  for  student  testing,  ho.rever,  the  incident 
resulted  in  the  author's  loss  of  faith  in  the  value  of  reviews.  From  his 
standpoint,  reviewer  credibility  was  at  a fairly  low  level  for  the  remainder 
of  the  project. 

Not  all  authors  responded  in  this  manner,  however.  A number  of  authors 
made  the  comment,  "Often  when  you  say  something  in  a lesson  isn't  working, 
in  fact  the  students  object  to  it  too.  How  do  you  know  what  they'll  object 


19 


to?"  One  such  author  did  see  the  need  for  student  testing  to  validate 
lessons,  and  in  fact  went  to  great  lengths  to  recruit  students  to  go 
through  lessons  before  actual  course  students  started  going  through  the 
PLATO  course. 

Authors  also  reiterated  the  theme  "Our  students  are  different  than 
any  others  because  ..."  with  students  either  being  viewed  as  "too 
smart"  or  "too  dumb"  for  a suggested  approach.  Since  in-progress  reviewing 
was  honed  mainly  on  a paramedical  project,  the  variation  reviewers  heard 
most  frequently  was,  "Our  students  are  even  different  from  other  military 
students  because  they're  so  highly  motivated  that  all  you  have  to  do  is 
present  the  material  and  not  really  worry  too  much  about  how  you  do  it. 
Sophisticated  approaches  aren't  really  necessary  and  only  slow  them  down." 
When  three  of  the  original  sixteen  students  (an  alarming  rate  for  this 
paramedical  course)  flunked  out  of  the  paramedical  program  entirely  and 
the  remainder  of  the  students  asked  to  be  reassigned  to  the  regular  class- 
room, the  theme  became,  "These  students  are  so  poor  that  'sophisticated 
approaches'  are  beyond  them."  For  example,  a common  reviewer  comment  was 
that  questions  were  aimed  at  too  low  a level  of  learning,  requiring  the 
student  to  demonstrate  simple  recall  rather  than  apply  a concept  to  a 
new  situation.  Here  again,  review  comments  conformed  to  actual  Sheppard 
course  document  requirements  which  specified  the  need  for  teaching  to  and 
testing  at  the  highest  cognitive  levels  (analysis,  synthesis,  evaluation, 
etc.)  over  recall  or  restatement.  At  the  beginning  of  the  project,  sotie 
authors  claimed  that  "sophisticated  approaches"  took  up  the  student's 
valuable  time.  Later,  after  reviewers  again  made  the  same  objection  to 


a number  of  later  lessons,  the  same  authors  retorted  that  "these"  students 
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were  at  too  low  a level  for  "sophisticated"  questions,  and  had  to  be 

•I 

I "spoonfed"  the  material.  Some  authors  also  admitted  that  part  of  the 

problem  was  the  difficulty  in  coding  short  answer  types  of  questions, 
though  a full-time  programmer  was  on  staff  at  the  site. 

Over  the  course  of  the  project,  MTC  reviewers  observed  a number 

■ 

i of  author  behavioral  changes,  without  apparent  corresponding  attitudinal 

i : changes.  For  example,  one  author  consistently  resisted  writing  student 

I objectives  for  his  lessons.  In  a number  of  consecutive  reviews,  the 

f ■ 

i reviewer  reiterated  the  usefulness  of  specific  student  objectives  in 

assisting  the  author  in  knowing  what  material  to  include,  how  the  infor- 
mation should  be  organized,  what  he  actually  expected  from  the  student  and 

I 

how  he  was  going  to  test  whether  the  student  had  learned  what  he  intended, 
etc.  After  a while,  the  author  finally  began  to  write  objectives,  at 
times  even  submitting  objectives  for  review  apparently  before  the  lesson 
had  even  been  started.  The  reviewer  assumed  this  represented  a change 
(or  rather  an  addition  to)  the  author's  instructional  philosophy.  How- 
ever, when  it  was  pointed  out  that  the  objectives  and  content  of  one  of 
his  lessons  didn't  match  he  replied,  "Ok,  I'll  write  the  objectives  over 
again."  The  thought  that  it  was  simpler  to  just  rewrite  the  objectives 
showed  a lack  of  understanding  of  what  the  purpose  of  stating  objectives 
I really  was. 

! In  another  Instance,  an  author  and  reviewer  debated  at  some  length 

i the  value  of  interspersing  the  lesson  with  interactive  questions  in 

reinforcing  the  lesson  material,  giving  the  student  practice  at  utilizing 
information  he  would  later  be  tested  on,  "monitoring"  the  student's 
progress,  etc.  One  of  the  sources  of  disagreement  was  in  their  conflicting 
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definitions  of  "interaction".  The  reviewer  defined  interaction  as  any 
situation  in  which  the  student  is  required  to  respond  by  writing  a 
comment,  asking  a question  or  giving  an  answer.  The  author  defined  inter- 
action as  any  student  input  whatsoever  including  simple  keypresses  (NEXT, 
BACK,  etc.).  The  author  finally  began  to  put  "interaction"  into  his 
lessons,  but  since  his  definition  was  so  broad,  simply  Increasing  the 
number  of  student  inputs  did  not  appreciably  raise  the  quality  of  his 
lessons.  Also,  what  questions  he  did  include  were  mainly  yes/no  or 
copyframe  types  in  which  the  student  simply  scanned  the  page  for  the 
correct  answer  and  then  "recorded"  it.  In  some  cases,  questions  were 
not  even  at  the  level  of  simple  recall.  It  seemed  apparent  that  he  was 
including  interaction  strictly  for  Interaction's  sake. 

In  another  episode,  a number  of  authors  at  one  site  were  producing 
strictly  linear  lessons  with  all  students  routed  through  the  same  path. 
While  a linear  format  is  certainly  not  inherently  detrimental,  the 
reviewer  tried  to  persuade  authors  to  consider  including  at  least  a 
small  number  of  lessons  in  which  the  student  had  more  autonomy.  Indexes, 
"crossroad"  type  choice  pages,  etc.  were  suggested.  Soon  the  reviewer 
began  to  see  a flurry  of  index  pages  appear.  However,  students  were 
still  instructed  to  go  through  the  lesson  in  the  order  given.  In  one  case, 
while  a lesson  index  was  supplied,  it  seemed  in  fact  to  be  more  a table 
of  contents  since  the  student  was  never  allowed  to  choose  where  to  go. 

In  all  these  instances,  it  is  apparent  that,  while  external  changes 
were  certainly  made,  the  attending  internal  changes  never  crystallized. 

In  none  of  the  preceding  incidents  is  it  clear  whether  the  problem 
was  the  reviewers'  collective  lack  of  persuasive  ability,  lack  of 
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communication,  or  simply  that,  rather  than  debate  the  issues  any  longer, 
authors  chose  to  make  token  changes.  In  a number  of  instances  real 
changes  were  made.  For  example,  a number  of  authors  at  one  site  began 
to  eliminate  "copyframe"  questions  almost  entirely  from  their  lessons, 
learning  instead  to  write  questions  aimed  at  higher  levels  of  learning. 

In  many  cases,  however,  it  seemed  apparent  that,  though  authors'  approaches 
sometimes  changed,  the  attending  attltudinal  changes  never  metamorphosed. 

What  Can  Be  Accomplished 

In  spite  of  our  sometimes  frustrating  experiences,  we  feel  very 
strongly  that  some  sort  of  lesson  review  is  essential  not  only  for 
inexperienced  authors  but  also  for  already  established  authors.  In  the 
same  way  that  the  publishing  industry  exercises  editorial  standards, 

CBE  needs  some  quality  control  methods  to  insure  at  least  minimum  lesson 
standards.  Reviewing  can  be  a valuable  tool  for  lesson  development, 
a "soft"  step  between  programming  and  feedback  from  student  runs.  Based 
on  our  experiences  and  perceptions,  and  comments  from  authors  at  various 
sites,  we  feel  confident  in  making  the  following  recommendations.  While 
nearly  all  our  testing  and  theorizing  has  been  done  with  the  military, 
many  suggestions  should  apply  with  equal  validity  in  any  number  of  situa- 
tions. 

After  compiling  and  sorting  through  past  recollections  and  comments, 

three  major  areas  of  concern  standout: 

— reviewer  characteristics 
— reviewer/author  relationship 
— review  content/format 


i 


The  following  discussion  deals  with  suggestions  in  all  three  areas. 
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Reviewer  Characteristics 

i 

In  a resounding  expression  of  unanimity,  all  authors  agreed  that  a 

i 

•>  lesson  reviewer  should  be  on  site.  This  is  perhaps  the  strongest  and  most 

widely  expressed  recommendation  we  will  make.  As  one  author  wrote. 

Of  all  the  reviews  that  I had,  I felt  the  most  useful  one  was 

^ the  one  which  I had  side  by  side  with  when  she  was  down 

, here.  I felt  I learned  more  in  that  2-3  hour  period  than  in 

all  the  other  reviews.  There  were  alot  of  things  she  said  and 
did  that  just  wouldn't  have  been  included  in  a review.  Reviewers 
; should  have  come  down  to  deliver  the  reviews  personally  or  to 

, do  the  reviews  right  there.  I'm  not  sold  on  the  idea  that  you 

can  do  reviews  long  distance. 

Some  authors  also  felt  that  a reviewer  should  have  teaching  experience, 
though  there  was  wide  disagreement  over  whether  the  reviewer  should  also  be 
a subject-matter  expert.  Most  authors  seemed  to  think  that  subject- 
matter  expertise  was  not  necessary.  One  author  even  said,  "Sometimes 
it  was  good  that  the  reviewer  didn't  have  subject-matter  experience.  It's 
too  easy  [for  a subject-matter  expert]  to  miss  some  things  ...  an  outsider 
can  ask  'what's  going  on  here?'"  Since  most  sites  are  generally  staffed 
with  a number  of  subject-matter  authors,  we  feel  no  real  need  for  the 
reviewer  to  be  a subject-matter  specialist,  too.  A number  of  non-ARPA 
sites  (as  well  as  one  notable  ARPA  site)  have  observed  the  tendency  for 
subject-matter  experts  to  "review"  lessons  solely  on  the  basis  of  content 
■ accuracy.  In  fact,  some  projects  and  authors  specifically  requested 

1 subject  matter  reviews  only,  stating  explicitly  that  comments  on  presenta- 

tion, Instructional  strategy,  etc.  are  of  no  interest.  This  is  certainly 
a valid  area,  but  by  no  means  the  breadth  of  what  a lesson  is  or  the  only 
criterion  for  lesson  effectiveness.  Though  some  content  specialists 
would  disagree,  content  expertise  does  not  presuppose  Instructional  exper- 
tise. By  the  same  token,  lack  of  content  experience  need  not  imply  lack 

I 


I ' of  credibility.  A lesson  needs  to  be  critiqued  from  a number  of  different 

perspectives,  one  of  which  is  from  the  instructional  design  point  of  view. 

i «. 

’ Thus,  just  as  it's  reasonable  to  ask  a content  expert  to  review  the  content, 

it's  also  reasonable  to  rely  on  an  instructional  designer  to  review  the 
instructional  approach. 

A more  illusive  but  equally  important  quality  is  reviewer  personality. 
Nearly  all  authors  agreed  that  a reviewer  must  be  able  to  "handle"  a 

t ' 

I 

. , variety  of  author  temperaments.  Typical  author  comments  are,  " . . . it's 

very  tough  to  find  the  right  individual  who  can  handle  each  author  indivi- 
dually and  appropriately",  and  "Personality  requirements  vary  with  author 

* temperament."  A reviewer  needs  to  be  something  of  a psychologist,  able 

to  employ  various  approaches.  For  a complete  description  of  review 
techniques  used  throughout  the  MTC  project  see  Lesson  Review  (Francis, 
Goldstein  & Call-Hirawlch , 1975). 

Reviewer/Author  Relationship 

Many  authors  (and  reviewers)  felt  that  the  success  or  failure  of  a 
review  was  a direct  result  of  the  relationship  between  author  and  reviewer. 
As  one  author  said,  "It's  all  so  personal.  It’s  all  in  the  relationship 
between  the  reviewer  and  author."  Another  author  stated,  "Rapport  is 
important  . , . it's  intangible,  but  it  effects  the  results."  Rapport 
will  always  be  a factor  in  any  cooperative  effort.  However,  the  heavy 
reliance  on  rapport  in  reviewing  is  often  counterproductive.  A number  of 
steps  can  be  taken  to  reduce  the  importance  of  the  subjective  relationship 
between  author  and  reviewer  and  to  clarify  the  purpose  of  the  review 
itself.  First,  if  the  reviewer  is  to  be  in  any  way  responsible  for  the 

i , 
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quality  of  the  lessons  produced,  (s)he  should  be  accorded  some  authority 
at  the  outset  by  the  funding  agency  or  project  management.  A "consultant" 
position  is  not  a strong  vantage  point  from  wliich  to  effect  change.  In 
any  project  (particularly  in  the  military),  if  an  on  site  reviewer  is  a 
member  of  the  military,  his/her  rank  should  be  equal  to  that  of  the  highest 
ranking  author.  At  one  ARPA  site,  the  only  instructional  design  specialist 
was  an  enlisted  man  while  all  but  one  of  the  remaining  authors  were  officers. 
His  suggestions  were  not  well  received,  and  were  in  fact  either  totally 
ignored  or  never  requested  at  all.  If  the  reviewer  (either  on  or  off  site) 
is  a civilian  working  with  military  authors,  his/her  perceived  authority 
must  be  equal  to  that  of  the  authors.  This,  of  course,  contains  the  seeds 
of  a reverse  sort  of  problem.  If  the  reviewer  has  higher  authority,  some 
authors  may  follow  the  letter  rather  than  the  spirit  of  suggestions  to 
give  the  impression  of  obeying  a superior.  In  the  long  run,  the  most 
important  factor  may  still  be  the  relationship  between  author  and  reviewer. 

If  the  reviewer  is  off  site,  his/her  credibility  can  be  greatly 
enhanced  by  frequent  site  visits  at  which  times  (s)he  can  become  familiar 
with  authors  problems  in  the  environment  in  which  they  must  function.  This 
would  also  help  in  dispelling  the  "ivory  tower"  syndrome.  For  example, 
two  authors  at  one  ARPA  site  repeatedly  discounted  portions  of  what  MTC 
reviewers  suggested,  saying,  "You  don't  understand.  You  don't  understand 
our  students  or  our  problems."  When  a reviewer  finally  visited  the  site, 
both  authors  commented,  "Now  that  you've  seen  our  students,  you  won't  have 
any  trouble  reviewing",  though  the  reviewer  had  in  fact  only  literally  "seen" 
the  students  in  hallways,  etc.  rather  than  in  a scheduled  PLATO  class. 

Though  the  reviewer  felt  there  was  little  dlscemable  difference  in  the 
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comments  she  made  in  successive  reviews,  both  authors  seemed  to  be 
satisfied  that  now  her  reviews  were  "better".  How  much  credibility 
exists  in  fact  and  how  much  in  the  eye  of  the  beholder  is  sometimes 
speculative.  If  the  problem  exists,  however  (and  it  usually  does), 
debating  the  point  will  never  solve  it. 

A second  step  toward  clarifying  the  purpose  of  the  review  would  be 
to  contract  at  the  outset  for  specific  feedback  the  author  would  like 
from  a review.  This  is  particularly  important  if  the  reviewer  is  seen 
only  as  a consultant.  To  facilitate  this  and  to  establish  some  common 
instructional  ground,  the  author  and  reviewer  should  also  meet  before 
even  the  first  review  is  requested  and  try  to  establish  an  Instructional 
design  concensus.  This  would  also  afford  each  an  opportunity  to  get  a 
better  feeling  for  the  other's  biases,  nomenclature,  etc. 

Review  Format /Content 

One  of  the  most  important  recommendations  we  could  make  is  the 
establishment  of  on-site,  peer  reviews  as  a regular  part  of  the  workings 
of  a project.  This  could  take  the  form  of  regular  group  meetings,  or 
individual,  more  informal  one-on-one  types  of  reviews.  At  every  site  at 
which  peer  reviews  were  a regular  routine,  authors  said  they  relied 
approximately  half  on  MTC  reviews  and  half  on  peer  reviews  both  for 
feedback  and  as  a tool  for  revision.  In  order  to  effectively  review 
each  other's  lessons,  however  (l.e.  comment  on  more  areas  than  spelling, 
punctuation,  content  accuracy,  etc.),  authors  also  need  more  explicit 
training  in  instructional  design. 

If  possible,  reviews  should  always  be  performed  face-to-face  with  the 
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reviewer  involved  in  the  lesson  from  the  planning  stages  on.  If 
reviews  can't  be  done  in  person,  on-line  monitoring  or  even  over-the-phone 
reviewing  are  good  alternatives  and  preferable  to  end-of-lesson  or 
hard-copy-only  type  reviews. 

A lesson  should  also  be  reviewed  in  segments.  We  found  this  to  be 
a highly  effective  deterent  to  the  buildup  of  author  defensiveness  or 
subj activity. 

Finally,  depending  on  the  contracted  specific  feedback,  reviews 
should  stress  organization,  alternate  instructional  approaches,  clarity 
of  text,  quality/quantity  of  interaction,  etc.  Since  enumerating  mis- 
spellings and  grammatical  errors  both  is  time  consuming  and  increases 
the  number  of  perceived  "criticisms"  in  a lesson  review,  the  lesson  should 
already  have  been  proofread  for  such  incidentals  before  a review  is 
requested. 
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Appendix  A:  Author  Attitude  Questionnaire 


The  following  is  a list  of  suggestions  you  may  or  may  not  have  been 
looking  for  in  an  MTC  review.  Please  indicate  the  importance  of  those 
items  you  WANTED  by  using  the  rating  scale  provided.  Items  you  were 
NOT  Interested  should  rate  as  0. 


4 — wanted  and  very  important 
3 — wanted  and  important 
2 — wanted  and  moderately  important 
1 — wanted  and  minimally  important 
0 — NOT  wanted 

a.  grammatical  usage,  spelling  punctuation,  etc. 

b.  quality/quantity  of  student  Interaction 
_c.  appropriateness  of  tone  and  style 

_d.  clarity  of  explanations 

_e.  ef fectiveness/arrangeraent  of  graphic  displays 

f.  content  organization 

g.  content  accuracy 

_h.  alternate  instructional  strategies 

_i.  coding  efficiency 

J.  other  (if  other,  please  specify) 
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2.  Using  the  following  scale,  please  rate  the  importance  in  lesson 
development  of  the  sources  of  information  listed  below. 


4 — very  important 
3 — moderately  important 
2 — minimally  Important 
1 — not  important 
0 — don't  know 

a,  local  review  by  project  member(s) 

_b.  review  by  MTC 
_c.  review  by  instructor/users 
_d.  review  by  outside  experts 
e.  observation  of  students 

_f.  comments  or  questionnaire  responses  from  students 
_g.  student  data  (on-line) 
h.  test  or  quiz  results 
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3.  What  were  the  main  disadvantages  of  MTC  reviews? 

i 


1 4.  What  were  the  main  advantages  of  MTC  reviews? 

I 

I 


5.  I regarded  the  MTC  reviews  I received  as: 

a.  Generally  a waste  of  time 

b.  Interesting  but  not  very  useful 

c.  Moderately  useful 

d.  Very  useful 

6.  In  what  way,  if  any,  did  you  change  your  approach  to  lesson  writing 
based  on  the  information  you  received  in  MTC  reviews? 


7.  How  would  you  characterize  the  "ideal"  reviewer  with  respect  to: 
subject-matter  knowledge,  personality,  geographical  location, 
teaching  experience,  instructional  design  experience,  etc.? 
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